Sharing data between 2 PagerDuty tenants

#1

Our company has 2 PagerDuty tenants; one in the UK and the other in Spain. The challenge we have is that one of the support teams is based in Spain and therefore will be configured in the Spanish tenant. However in certain scenarios the Spanish support team will need to be engaged from the UK PD tenant (either by way of pre-configured Response Plays or manually added as Responders to an Incident) in order to assist with particular problems on a UK-based service.

So we need the Incident to be created in the UK tenant, but notify the support team in Spain as one of the Responders. Then when the Spanish support team “accept” the invite to join, or have any updates we’d like this information to be populated in the Incident in the UK tenant.

Can anyone advise whether this achievable at all please? The 2 tenants within our company are completely different entities and the intention is to keep them that way, with the Spanish users in their tenant and not in the UK one.

Many thanks
Chris

#2

I recommend using something in the middle (AWS Lambda, Azure Function, etc) to receive webhooks from tenant 1 and process them into corresponding actions (new incident, run response play, add responders, etc.) in the other tenant.

1 Like
#3

Thanks Doug.

And would it be feasible to have this piece of middleware receive webhooks from tenant 2 in order to update the original Incident in tenant 1 with either Notes on the Incident or even resolve it?

Regards
Chris

#4

Yep, you can get webhooks for nearly everything, have them processed in the middleware, then ignored or routed to the other tenant as needed by type/source. Look to webhooks v3 for more granular options.

1 Like
#5

Bit of relevant trivia here: one reason that one would need to use a Lambda or some other third party service to receive webhooks from one PagerDuty account before translating them to events inbound for another PagerDuty account is that PagerDuty doesn’t permit webhooks to go through its event processing pipeline. Otherwise, it would be trivial to create a feedback loop by accident (or otherwise).

That being said, please be cautious when doing this sort of thing. Also, I’d be remiss if I didn’t mention: if by “tenant” it is meant two separate teams in the same PagerDuty account, there are easier ways of implementing this :wink:

#6

Thank you for your response Demitri. This subject is a bit beyond my level of expertise but will take the advice onboard and will share the information in discussions internally.

I am referring to 2 separate accounts, owned by different geographical areas of our global organisation.

Regards
Chris

#7

Hello Chris,

Thanks for that clarification.

Let us know if we can be of further assistance.

Kind regards,

#8

Hi there o/

I’m in the same position where we have x2 separate orgs which have their own instances of PagerDuty.

Org B would have service dependencies in Org A.

What I’ve played with (and works!) is using Webhooks v3 in Org A to send “Service Foo” incident payloads to a custom PagerDuty App Event Transformer in Org B, which I then transform and push to a dummy “Service Bar” service again in Org B·

What I’d like is to run that Transformer myself, Lambda I guess, and have one endpoint where I can send multiple Org A service webhooks to forward to Org B Event Rules in PD-CEF format and do the logic there.

Is there any sample code for that Lambda function or pointers?

I realize I’d have to handle multiple webhook secrets, or maybe basic auth over https will be enough.

#9

Do you think you’d use Python or Node.js Lambda?

1 Like
#10

Hi Doug,

Sorry for delay.
Either one, maybe Python?

I may look at Event Bridge too. What I’ll end up doing is documenting this internally to unblock other teams.

If on the other hand all webhooks can be sent to the one endpoint, my team will probably “own” the Transformer/Lambda/Event Bridge setup, whatever it ends up looking like.

#11

Hey @dmcclure :wave:

Do you have some sample code I could use?

#12

See if this helps you see how this could be done. Source PD Instance would send Webhook V2 (you should really use Webhook v3) to AWS Lambda that would parse the trigger webhook and build a new incident payload to create an incident in the target PD instance on a given service. You could get much more elaborate with how you evaluate the incoming webhook and decide what service to send it to in target, what priority, what EP to use, etc.

NOTE: I didn’t test this code!

#!/usr/bin/env python
from botocore.vendored import requests
import json

#  use env variables or KMS

PD_API_key = "FOO" #target RW API token
API_user = "FOO@BAR.COM" #target user email for inc creation

def createIncidentInTarget(serviceID,priorityID,incID,incTitle,incBody):
	headers = {'Content-Type': 'application/json', 'Accept': 'application/vnd.pagerduty+json;version=2',
			   'From': API_user, 'Authorization': 'Token token=' + PD_API_key}

	payload = createIncidentPayload(serviceID,priorityID,incID,incTitle,incBody)
	endpoint = 'https://api.pagerduty.com/incidents'

	resp = requests.post(endpoint, headers=headers, data=json.dumps(payload))

	json_resp = resp.json()

	print(json_resp)

	return json_resp


def createIncidentPayload(serviceID,priorityID,incID,incTitle,incBody):
	payload = {
	  "incident": {
		"type": "incident",
		"title": incTitle, #source inc title
		"service": {
		  "id": serviceID, #target serviceID
		  "type": "service_reference"
		},
		"priority": {
		  "id": priorityID, #target priorityID
		  "type": "priority_reference"
		},
		"urgency": "high",
		"incident_key": incID, #source-serviceID-source-incID
		"body": {
		  "type": "incident_body",
		  "details": incBody #source inc body
		}
	  }
	}
	return payload

def lambda_handler(event, context):

	# trigger a new incident in target based on source inc webhook data

	if event['detail']['event'] == "incident.trigger":  #only for trigger incidents from source

		# source incident ID
		incID = event['detail']['incident']['id']

		# source incident title
		incTitle = event['detail']['incident']['summary']

		# source incident body
		incBody = event['detail']['log_entries']['channel']['details']

		# target service ID
		serviceID = "P12345"

		# target priority ID
		targetP1 = "P98765"
		targetP2 = "P99887"

		if event['detail']['incident']['priority']['summary'] == 'P1':
			createIncidentInTarget(serviceID,targetP1,incID,incTitle,incBody)

		if event['detail']['incident']['priority']['summary'] == 'P2':
			createIncidentInTarget(serviceID,targetP2,incID,incTitle,incBody)

	return 1
1 Like
#13

Excellent!
I’ll give this a try first thing in the am.
Thanks Doug

#14

Hey Doug,

What I’m hoping to get is where we use Webhook v3 from multiple services in source PD org and send these to one Lambda endpoint.
From there we convert the payload to a PD-CEF one and forward it to the Target PD instance via Global Event Rule. We’d do the logic in the RuleSet. The idea is we’ll have “dummy” services setup where folks can gets a visual when it has an incident (in the Source PD org) and add dependencies etc.

The webhook would fire on recovery too and be reflected again the the Target PD dummy service.

I have this working using PD App Event Transformer but it would mean one app per Source service, 1-1 relationship.

export function transform(PD) {
    // capture incoming webhook body
    let body = PD.inputRequest.body;
    // capture incoming webhook headers (should we need them)
    const headers = PD.inputRequest.headers;
    let emitEvent = true;
   
    // Incident stuff
    let event_type = PD.Trigger;
    let incident_key = body.event.data.id;

    // Check event_type value for acknowledged or resolved
    if(body.event.data.status == "acknowledged") { event_type = PD.Acknowledge; }
    if(body.event.data.status == "resolved") { event_type = PD.Resolve; }

    // Preparing the normalized_event to be sent to pagerduty
    let normalized_event = {
        event_action: event_type,
        dedup_key: incident_key,
        description: body.event.data.title,
        details: body.event.data.service.summary,
        payload: {
            summary: body.event.data.service.summary,
            source: body.event.data.html_url,
            severity: PD.Critical,
            custom_details: `PD.inputRequest.body PD.inputRequest.uri`
        },
   
        links: [{
            "href": body.event.data.html_url,
            "text": "View In PagerDuty"
        }],
    }

    if (emitEvent) PD.emitEventsV2([normalized_event])
  }
#15

You could totally do what you described! In the Lambda, you would need to parse the incoming webhook payload then reconstruct it into an Event API v2 (PD-CEF) formatted event and send it to the Event API v2 endpoint with an event ruleset token. Then you can route the events to the desired target services from there.

One current gap in Webhook v3 is you won’t have access to the raw event data from the source until later this year. You could use Webhook v2 today and get the full raw event data in the trigger webhooks if you needed that to recreate the event in detail for the target.

#16

I’ve opened a support ticket for this one Doug :+1:

#17

Just a quick followup, this is the python Lambda code I settled on and seems to work quiet well.
This might help other folks out there.:vulcan_salute:

#!/usr/bin/env python3
import os

import requests
import json

# Main Lambda Entrypoint.

URL = os.environ['URL']
ROUTING_KEY = os.environ['ROUTING_KEY']

def lambda_handler(event, context):
body = json.loads(event["body"])

print(body['event'])
response = {}

# Verify event_type exists in payload, throw a 500 otherwise.
if "event_type" in body['event']:
  print ("Processing PagerDuty Generic Webhook (v3) Payload")
  print ("The Event type is " +  body['event']['event_type'])
  
  # Set the incident type from payload (event_type)
  if body['event']['event_type'] == 'incident.triggered':
     EVENT_ACTION = "trigger"

  if body['event']['event_type'] == 'incident.acknowledged':
     EVENT_ACTION = "acknowledge"

  elif body['event']['event_type'] == 'incident.resolved':
     EVENT_ACTION = "resolve"

  # Define PD API config.
  pd_event_api = URL
  pd_headers = {
      "Content-Type": "application/json"
  }

  # Pass through alert into PagerDuty with some conformation to PD-CEF.
  pd_event = {
      "event_action": EVENT_ACTION,
      "client": body['event']['client'],
      "routing_key": ROUTING_KEY,
      "dedup_key": json.dumps(body['event']['data']['number']),
      "payload": {
          "summary": body['event']['data']['title'],
          "severity": "info",
          "source": body['event']['data']['html_url'],
          "custom_details": body['event']
      }
  }
  pd_event_response = requests.post(
      url=pd_event_api, headers=pd_headers, json=pd_event)

  # Update response based on PD API.
  response = {
      "status_code": pd_event_response.status_code,
      "response_body": json.dumps(pd_event_response.json())
  }

  # Update response if params are missing.
  else:
    response = {
    "status_code": 500,
    "response_body": "Parameter(s) missing from endpoint",
    "queryStringParameters": body['event']
}

  print(response)
  return response

lambda_function.py.txt (2.0 KB)